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19.  Abstract  (continued) 


development  of  a  bloqrbernetlc  device,  such  a  device  would  employ  theau  suitable 
blumedlcal  measures*  analyze  them  In  real  time,  and  feed  the  Information  back  to  the 
mchlne  (or  operator)  in  order  to  modify  systems  performance. 

For  the  present  Phase  I  report*  three  separate  efforts  were  conducted.  The  first* 
a  pilot  study,  employed  five  subjects  to  whom  performance  tests  were  administered 
while  the  feasibility  of  the  eye  movement  recording  techniques  was  explored  and  to 
gain  experience  with  the  scoring  procedurbs.  Then*<  In  the  main  experiment*  15 
subjects  were  exposed  to  three  different  levels  of  workload  difficulty  over  six 
sessions  (each  workload  condition  repeated  twice).  Performance  (percent  correct)  on 
the  tasks  and  eye  movement  metrics  (blink  rate,  frequency*  amplitude*  and  velocity  of 
saccades)  were  recorded  throughout.  In  addition  to  the  task  difficulty*  "time  on 
task*  was  examined  as  a  dependent  variable  on  bioelectric  and  performance  metrics.  In 
a  third  study*  longer  sessions  were  employed  to  gain  experience  with  the  effects  of 
sustained  performance  on  the  eye  movement  and  performance  metrics. 
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In  the  main  experiment*  It  was  demonstrated  that  group  performance  (percent 
correct)  on  the  tone  counting  task  varied  with  task  difficulty  (p  <  .01)*  suggesting 
that  the  workload  had  been  successfully  manipulated.  Of  the  eye  movement  measures* 
acceleration  of  eye  movements  (or  the  slope  of  the  regression  line  relating  velocity 
of  saccades  to  amplitude)  bore  a  strong  relationship  to  task  difficulty*  becoming 
steeper  (faster  eye  movements)  for  80%  of  the  subjects  (p  <  .01)  when  the  high  task 
loading  condition  was  compared  to  the  lower.  Bye  movement  frequency  and  eye  blinks 
did  not  appear  to  be  different  In  the  two  task  loadings  (p  •  .40).  For  the  ether  eye 
movement  measures  explored:  (a)  amplitude  of  saccades  tended  to  Increase  In  the  high 
task  load  condition  for  the  majority  of  the  subjects  (61%)  but  this  difference  was  not 
significant  (p  <  .15)*  and  (b)  aggregated  eye  movement  velocities  under  the  high 
workload  were  generally  greater  In  the  high  workload  condition  for  the  majority  of  the 
subjects  (73%;  p  <  .02). 

While  the  relationship  between  time  on  task  and  performance  followed  predicted 
relationships*  the  size  of  the  effect  was  small  (4%)  and  not  statistically  significant 
(p  <  .20).  Although  performance  may  not  have  degraded  under  the  conditions  as  were 
measured,  bioelectric  measures  nonetheless  were  predictive  of  time  on  ta'ik:  (a) 
frequency  of  eye  movements  declined  for  73%  (p  <  .037)  of  the  subjects;  and  (b) 
acceleration  of  eye  movements  Increased  for  66%  (p  «  .098)  suggesting  that  both 
metrics  hold  promise  as  prospective  measures  of  sustained  performance. 

Finally*  as  part  of  the  present  effort*  we  also  have  developed  considerable 
customized  computer  techniques  and  algorithms  for  rapid  scoring  and  analysis  of  these 
key  eye  movement  metrics.  Presently*  60%  of  all  scoring  for  the  eye  movement  activity 
system  Is  fully  automatic  (l.e.*  eye  blink  and  saccade  metrics)  and  the  remainder  Is 
conducted  Interactively  using  an  experimenter.  This  reduces  by  90%  the  scoring 
process  for  the  experimenter  and  was  seen  as  a  necessary  intermediate  step  In  the 
development  of  a  fully  automated  scoring  system.  We  believe  there  are  no  technical 
Impediments  to  Improving  the  automatic  scoring  techniques  so  that  analyses  could  be 
conducted  In  near  real  time  with  more  sophisticated  equipment. 
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Workload  measuramant  involvas  an  attampt  to  charactarlza  conditions 
under  «diich  task  demands  can  or  cannot  be  mat  by  the  performer.  In  the 
early  days  of  aviation*  the  machine  was  often  the  limiting  factor  in  system 
performance.  Today,  overstressed  aviators  may  be  more  common  than 
overstressed  aircraft.  New  systems  permit  more  information  to  be  presented 
in  real-  or  fast-time  to  the  operator  than  he  or  she  can  handle 
efficiently.  In  modern  weapon  systems  the  human  sensory  systems  are 
overloaded  and  the  problem  is  not  likely  to  abate.  While  military  equipment 
is  becoming  more  complex,  the  proportion  of  the  available  pool  of  personnel 
which  possesses  the  mental  capabilities  to  accommodate  these  emerging 
systems  is  becoming  more  limited  (Tice,  1986;  Merrlman  &  Chateller,  1981). 

There  are  many  ways  in  vdiich  workload  may  be  studied.  The 
investigations  presented  here  presitme  that  biological  events  may  be 
predictive  of  the  task  demands  on  the  operator  as  well  as  the  individual's 
personal  attention  to  that  work.  For  example,  we  know  there  may  be  little 
or  no  deterioration  in  operational  performance  until  the  point  of  failure  is 
closely  approached,  but  perhaps  sensitive  biological  measures  of  workload 
could  provide  premonitory  signs  of  impending  failure.  However,  to 
Investigate  such  a  set  of  relations,  we  believe  that  "workload”  needs  to  be 
measured.  But  before  it  can  be  measured,  it  needs  to  be  defined.  For  this 
purpose  we  have  adopted  a  model  presented  in  a  previously  conducted  projecc 
sponsored  by  the  U.S.  Air  Force.  In  that  work,  we  distinguished  between 
workload  measures  which  seek  to  characterize  the  stimulus  (we  call  them 
operator-independent  measures)  and  those  which  may  be  different  for 
different  subjects  (we  call  them  operator  dependent).  We  believe  it  is 
possible  to  have  objective  measures  (in  this  case  bioelectric  events  and 
performance)  which  can  index  both  constructs.  We  recognize  that  logically 
there  can  also  be  subjective  measures  of  both  of  these  constructs,  but  such 
Instruments  are  not  a  part  of  this  work. 

For  the  present  investigation,  we  systematically  varied  a  workload  task 
over  a  range  of  difficulties  and  sessions  while  eye  movement  elements  and 
performance  served  as  dependent  variables.  The  objective  was  to  determine 
whether  task  demands  which  are  inherent  in  the  stimulus  (e.g.,  number  of 
channels  monitored;  time  on  task)  covaried  with  characteristics  of  the 
dependent  variables.  Although  the  approach  was  empirical,  the  elements 
selected  for  study  followed  from  theory  or  findings  reported  previously  in 
the  scientific  literature.  The  Immediate  goal  of  this  work  was  to  surface  a 
series  of  objective  indicators  (both  performance  and  bioelectric  events) 
which  vary  systematically  with  workload  (both  operator  dependent  and 
operator  independent).  The  intermediate  goal  would  be  to  develop  a  battery 
of  measures  and  automated  scoring  algorithms  to  index  the  different  forms  of 
workload  effects  and  operator  state.  The  long-term  goal  of  this  work  is 
development  of  a  biocybernetlc  device.  Such  a  device  would  employ  these 
suitable  biomedical  measures,  analyze  them  in  real  time,  and  feed  the 
information  back  to  the  machine  (or  operator)  in  order  to  modify  systems 
performance. 

For  the  present  Phase  I  report,  three  separate  efforts  were  conducted. 
The  first,  a  pilot  study ^  employed  five  subjects  to  whom  performance  tests 
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wart  adialniftarad  whlla  the  faaflbllity  of  the  oya  movament  racordlng 
tachnlqusa  was  explored  and  to  gain  experience  with  the  scoring  procedures. 
Then,  In  the  main  experiment,  15  subjects  wore  exposed  to  three  different 
levels  of  workload  difficulty  over  six  sessions  (each  workload  condition 
repeated  twice).  Performance  (percent  correct)  on  the  tasks  and  eye 
movement  metrics  (blink  rate,  frequency,  amplitude,  and  velocity  of 
saccades)  were  recorded  throughout.  In  addition  to  the  task  difficulty, 
"time  on  task”  was  examined  as  a  dependent  variable  on  bioelectric  and 
performance  metrics.  In  a  third  study,  longer  sessions  were  employed  to 
gain  experience  with  the  effects  of  sustained  performance  on  the  eye 
movement  and  performance  metrics. 

In  the  main  study.  It  was  demonstrated  that  group  performance  (percent 
correct)  on  the  tone  counting  task  varied  with  task  difficulty  (£  <  .01), 
suggesting  that  the  workload  had  been  successfully  manipulated.  Of  the  eye 
movement  measures,  acceleration  of  eye  movements  (or  the  slope  of  the 
regression  line  relating  velocity  of  saccades  to  amplitude)  bore  a  strong 
relationship  to  task  difficulty,  becoming  steeper  (faster  eye  movements)  for 
80%  of  the  subjects  (£  <  .01)  when  the  high  task  loading  condition  was 
compared  to  the  lower.  Bye  movement  frequency  and  eye  blinks  did  not  appear 
to  be  different  In  the  two  task  loadings  (£  •  .40);  however,  other  eye 
movement  measures  were  explored:  (a)  amplitude,  the  SLIT  metric  of  previous 
APOSR  work,  tended  to  show  greater  amplitude  In  the  high  task  load  condition 
for  more  subjects  but  this  difference  was  not  significant  (61%;  £  <  .15); 
also,  and  (b)  aggregated  eye  movement  velocities  under  the  high  workload 
were  generally  greater  for  more  subjects  (73%;  £  <  .02). 

While  the  relationship  between  time  on  task  and  performance  followed 
predicted  relationships,  the  size  of  the  effect  was  small  (4%)  and  not 
statistically  significant  (£  <  .20).  Although  performance  may  not  have 
degraded  under  the  conditions  as  were  measured,  bioelectric  measures 
nonetheless  were  predictive  of  time  on  task:  (a)  frequency  of  eye  movements 
declined  for  73%  (£  <  .037)  of  the  subjects;  and  (b)  acceleration  of  eye 
movements  Increased  for  66%  (£  <  .098)  suggesting  that  both  metrics  hold 
promise  as  prospective  measures  of  sustained  performance. 

In  summary,  a  proof-of-concept  effort  was  conducted  in  order  to 
determine  whether  characteristics  of  workload  and  arousal  could  be  examined 
behavlorally  (percent  correct)  and  blomedlcally  (eye  movement  measures).  It 
was  found  that  a  performance  test  as  an  independent  variable  served  as  an 
exteroceptive  stimulus  (task  control  of  performance)  and  certain  eye 
movement  characteristics  were  surfaced  which  covaried  with  task  complexity. 
Other  related  eye  movement  metrics  also  showed  promise.  The  strength  of 
relationship  for  an  Interoceptive  stimulus  (viz.,  time  on  task)  was  weaker, 
but  eye  movement  metrics  were  surfaced  which  appeared  to  change  over  short 
sustained  periods.  Longer  periods  of  exposure  to  performance  tests  during 
eye  movement  recording  were  demonstrated  successfully  in  a  few  subjects  In 
order  to  verify  the  feasibility  of  this  procedure  for  sustained  performance 
experimentation. 

Finally,  as  part  of  the  present  effort,  we  also  have  developed 
considerable  customized  computer  techniques  and  algorithms  for  rapid  scoring 
and  analysis  of  these  key  eye  movement  metrics.  Presently,  60%  of  all 
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scoring  for  the  eye  movement  activity  system  is  fully  automatic  (i.e.,  eye 
blink  and  saccade  metrics)  and  the  remainder  is  conducted  interactively 
using  an  experimenter.  This  reduces  by  90%  the  scoring  process  for  the 
experimenter  and  was  seen  as  a  necessary  intermediate  step  in  the 
development  of  a  fully  automated  scoring  system.  Ve  believe  there  are  no 
technical  impediments  to  improving  the  automatic  scoring  techniques  so  that 
analyses  could  be  conducted  in  near  real  time  with  more  sophisticated 
equipment.  However,  we  believe  it  is  first  necessary  to  determine  what 
biomedical  elements  to  score.  To  this  end,  we  believe  this  system  can 
provide  an  important  adjunct  to  our  Automated  Performance  Test  System  (APTS) 
and  can  have  broad  implications  for  behavioral  performance  assessment.  For 
example,  all  of  the  APTS  tests  are  automatically  scored  and  would  permit  the 
biomedical  study  of  a  fully  researched,  factorlally  diverse  cognitive  and 
motor  battery  of  performances,  coincidentally  with  biomedical  relations  like 
eye  movements  to  which  they  may  be  differentially  related. 

II.  INTRODUCTION 

In  the  early  days  of  aviation,  the  machine  was  often  the  limiting  factor 
in  system  performance.  Today,  overstressed  aviators  may  be  more  common  than 
overstressed  aircraft,  and  recently  reported  Stealth  crashes  (Orlando 
Sentinel.  June  6,  1989)  imply  that  fatigue  following  heavy  schedules  were 
contributing  factors.  New  systems  permit  more  information  to  be  presented 
in  real-  or  fast-time  to  the  operator  than  he  or  she  can  handle  efficiently. 
Descriptive  terms  like  "getting  behind  the  system,"  "information  overload," 
and  "noisy"  abound  in  design  conferences,  and,  poignantly,  there  are 
"declutter  switches"  in  heads-up  displays.  In  modern  weapon  systems  the 
human  sensory  systems  are  often  so  bombarded  with  stimuli  that  Inferences 
from  displayed  information  are  time  constrained  and  performance  suffers. 
These  issues  are  not  limited  to  military  jobs.  Civilian  air  traffic 
controllers  are  often  retired  for  job* related  workload  stress  (Hale, 
Williams,  Smith,  &  Melton,  1971),  and  astronauts  experience  "time 
compression"  in  connection  with  their  high  workload  periods  upon  reentry 
(Schmitt  &  Reid,  1985).  As  military  equipment  has  been  made,  more  complex, 
the  proportion  of  the  available  pool  of  personnel  which  possesses  the  mental 
capabilities  to  accommodate  these  emerging  systems  is  becoming  more  limited 
(Tice,  1986;  Merriman  &  Chateller,  1981). 

Problems  caused  by  increasing  task  demands  do  not  come  as  any  surprise 
to  those  who  grew  up  in  the  field  of  human  factors  engineering.  Indeed, 
they  were  eloquently  predicted  by  Chapanls,  Garner,  Morgan,  and  Sanford 
(1947)  in  one  of  the  earliest  works  on  systems  research  where  they  said, 
"...Let  us  look  ahead  about  50  years  [it's  only  42!]  and  imagine  what  kind 
of  problem  the  air  traffic  control  officer  at  LaGuardia  Field  will  have  when 
a  heavy  fog  settles  over  the  place  and  100  planes  converge  on  the  field  from 
Mexico,  London,  Paris,  San  Francisco,  and  Albuquerque.  How  are  we  going  to 
get  all  that  information  to  him  [sic!].  How  can  we  present  this  information 
to  him  so  that  he  can  see  the  total  picture?  There  are,  after  all,  only  a 
few  channels  that  we  can  use  in  getting  this  information  into  his  brain"  (p. 
240).  Relatedly,  and  more  recently,  in  a  Congressional  hearing,  several 
prominent  human  factors  experts  from  different  fields,  inside  and  outside 
the  Department  of  Defense,  were  invited  to  offer  their  opinions  as  to  the 
cause  of  the  downing  of  the  Iran  Air  Flight  655.  Granted  that  an  incorrect 
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decision  had  been  made  from  displayed  Information,  the  one  issue  about  which 
the  experts  appeared  concordant  was  that  the  Incident  was  due  to  the 
combined  effects  of  the  high  workload  and  comliat  stresses  which  were  not 
accommodated  In  the  human  factors  design  of  the  tactical  Information 
displays  for  the  AEGIS  weapon  system  (House  Armed  Services  Committee,  1988). 

The  Idea  which  prompted  the  present  research  was  that  biological  events 
may  be  reflective  of  the  task  demands  of  the  work  as  well  as  the  operator's 
attention  to  the  task.  Ue  know  that  there  may  be  little  or  no  deterioration 
In  operational  performance  until  the  point  of  failure  is  closely  approached 
(Gopher  &  Donchln,  1986;  Schmidt,  1978),  but  perhaps  sensitive  biological 
measures  of  workload  could  provide  premonitory  signs  of  Impending  failure. 
There  are  two  technical  developments  which  must  be  accomplished  to  produce  a 
workable  system.  One  Is  traditionally  a  human  factors  effort,  and  In  the 
case  of  bioelectric  events.  Includes  several  aspects  of  the  biomedical 
sciences  and  technology.  The  second  Is  the  engineering  development. 
Including  software  and  hardware  which  must  accompany  such  an  enterprise.  In 
this  work  we  have  set  out  to  accomplish  both,  but  tie  emphasis  has  always 
been  on  the  former  rather  than  the  latter. 

There  are  many  applications  for  validated  biocybornetlc  workload 
indices.  To  the  extent  that  such  workload  measures  also  follow  task 
demands,  they  could  be  employed  to  Index  workload  characteristics  of 
military  systems  during  various  stages  In  the  human  factors  engineering 
design  and  subsequent  evaluation.  It  Is  generally  accepted  that  workload  Is 
a  function  of  task,  operator  capabilities,  and  sequential  characteristics 
(e.g.,  practice  with  task,  previous  workload  history,  etc.).  Moreover,  a 
general  consensus  Is  that  workload  Is  multidimensional  and  that  the 
researcher  should  have  a  battery  of  workload  metrics  at  his  or  her  disposal 
(Bateman,  1979;  Crabtree,  Bateman,  &  Acton,  1984;  Ecgemeler,  1980; 
Eggemeler,  Shlngledecker,  &  Crabtree,  1985;  Frazier  &  Cromble,  1982;  Gopher 
&  Donchln,  1986).  Researchers  are  determining  whether  there  are  reliable 
Indicants  of  Individual  differences  In  Information  processing  skills  (Benel, 
Coles,  &  Benel,  1979;  Damos,  1984a, b,c;  Damos  &  Smlnt,  1981;  Vlckens, 
Mountford,  &  Schreiner,  1981).  By  assessing  worklo«d's  Impact  on 
Individuals  for  tasks  of  constant  difficulty,  workload  Ind  .ces  can  be  used 
to  match  systems  according  to  the  reliable  individual  dl  .’ferences  of  the 
user  population.  Then  better  decisions  can  be  made  select lig  between  design 
tradeoffs  (e.g.,  customize  for  Individual  use,  design  In  "aided”  systems,  or 
automate  the  function).  Any  attempt  at  creating  a  blocy!)ei netic  system  will 
need  to  take  Into  account  the  demand  characteristics  of  he  workload  prior 
to  being  able  to  feedback  such  information  so  that  the  rsysten  produces  less 
workload. 

We  are  aware  that  there  is  a  need  to  do  better  e.t  w< ncload  measurement, 
but  we  would  suggest  that  the  Improvement  should  be  rot  only  from  the 
standpoint  of  the  workstation,  but  also  how  it  aff'.cts  the  Individual 
operator.  This  Is  not  to  say  that  we  believe  that  suojfjctlve  measures  of 
workload  are  necessary  but,  rather,  we  emphasize  that  ::hose  which  are 
subject  (or  operator)  dependent  should  be  studied  moce  extensively.  Our 
bias  is  that  the  physical  characteristics  of  the  stimi'lus  are  likely  to  be 
predictive  of  the  general  level  of  workload,  and  so  f?uitject  independent 
methods  of  workload  assessment  are  necessary.  However,  such  indices  are 
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chiefly  useful  to  create  specifications.  Because  so  much  of  the  reliable 
variance  in  operational  performance  Is  identifiable  as  being  due  to  the 
Individual  (Kennedy,  Jones,  &  Baltzley,  1988),  there  is  also  a  need  for 
subject  dependent  measures.  Note  that  both  of  these  (subject  dependent  and 
subject  independent)  can  be  objective  measures  (e.g.,  physiological 
indicants)  or  subjective  (e.g.,  self-report } .  We  believe  that  the  present 
study  will  deal  with  objective  measures  which  can  be  employed  for  both 
subject  dependent  and  subject  independent  assessment.  This  classification 
schema  and  the  implications  for  workload  measurement  are  described  in 
greater  detail  elsewhere  (Kennedy,  May,  Jones,  &  Fowlkes,  1989). 

Whatever  method  is  followed,  workload  metrics  are  needed  that  can  be  fed 
back  to  operators  to  remediate  performance  decrements.  This  paper  takes  the 
position  that  an  individual's  difficulty  with  a  work  station  —  vdiether  due 
to  tlme-on-task,  overload  or  some  other  factor  —  is  reflected,  perhaps 
unconsciously  by  some  neuroelectric  events  which  are  recoverable  and 
moasurable.  It  is  our  intention  to  capture  these  events  and  determine  ways 
in  which  they  can  be  assessed  portably  and  quickly  so  that  their  predictions 
can  be  fed  back  to  the  system  with  which  the  individual  interacts  in  order 
that  the  system  will  better  accommodate  the  individual's  needs.  Following 
Wiener  (1948),  we  think  that  something  like  this  is  what  K.U.  Smith  (1966, 
1967;  Smith  &  Smith,  1966)  had  in  mind  when  he  began  writing  about 
biocybernetics  some  25  years  ago. 

RBLATEO  RESEARCH 

Impetus  for  the  present  effort  began  several  years  ago  with  the 
experimental  observation  that  performance  on  a  vigilance  task  was  correlated 
with  patterns  of  eye  movements.  Specifically,  the  fast  phase  of 
vestibular-induced  nystagmus  appeared  to  disappear  as  performance  (percent 
correct)  degraded  over  a  sustained  period  of  time  (Kennedy,  1972).  This 
might  be  expected,  it  was  argued,  since  the  fast-phase  component  of 
nystagmus  is  dependent  on  the  integrity  of  the  reticular  formation  which  is 
also  related  to  arousal  and  alertness  (Cohen,  Feldman,  &  Diamond,  1969; 
Darhoff  &  Hoyt,  1971;  Yules,  Krebs,  &  Gault,  1966).  Given  that  other 
research  related  nystagmus  to  arousal  (Collins,  Crompton,  &  Posner,  1961; 
Collins  &  Posner,  1963),  the  idea  was  also  pursued  in  a  Navy  project  for 
divers  (Kennedy,  1978)  v/here  velocity  was  shown  to  be  related  to  both 
performance  and  time  on  task. 

More  recently,  under  sponsorship  of  AFOSR,  experimentation  of  an  eye 
movement  metric  of  workload  and  arousal  was  found  where  in  two  studies  (May, 
Kennedy,  Williams,  Dunlap,  &  Brannan,  1985)  subjects  performed  a  complex 
tone  counting  task  at  three  levels  of  difficulty  while  the  velocity  and 
latency  of  saccades  in  the  horizontal  plane  were  recorded  via  infrared 
reflection  techniques.  In  that  study  saccade  latency  appeared  to  vary  with 
the  task  demands  but  practice  effects  and  individual  differences  in 
performance  masked  the  relationship.  There  was  a  suggestion  that  the  length 
of  a  saccade  may  provide  information  about  task  demands.  Support  for  the 
"velocity  hypothesis"  was  not  found  •  -  saccade  velocities  obtained  during 
the  baseline  condition  of  free-viewing  did  not  differ  significantly  from 
velocities  obtained  under  tone  counting  conditions  • ■  however,  there  were 
known  limitations  in  sampling  rate  and  recording  sensitivity  of  the 
technique  employed. 
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In  the  second  project  (Kennedy,  May,  Jones,  &  Fowlkes,  1989)  the  same 
equipment  and  task  were  used  to  further  examine  the  relationship  of  eye 
movement  measures  (saccade  length)  to  workload.  The  general  finding  was 
that  the  mean  range  of  eye  movements  in  the  horizontal  plane  for  each  of  the 
three  workload  conditions  varied  inversely  with  difficulty  level  of  the  tone 
counting  task.  Thus,  the  results  from  this  research  suggested  that  saccade 
length  could  be  a  promising  objective  measure  of  the  task  demands  of  a 
display  and  thereby  serve  as  a  useful  measure  of  mental  workload. 

DESCRIPTION  OP  PROJECT  GOALS 

The  purpose  of  the  present  study  was  to  investigate  eye  movement  metrics 
which  would  covary  with  task  loading  in  order  to  derive  an  empirical  index 
of  workload.  Additionally,  because  such  metrics  are  likely  to  reflect  the 
task  demands,  it  was  predicted  that  changes  in  the  metrics,  vdiich  occur  over 
sustained  periods  of  time,  could  serve  to  signal  loss  of  attention  or 
vigilance.  An  ancillary  technical  obstacle  to  solution  of  such  problems  is 
the  requirement  for  specially  customized  scoring  and  analytic  techniques. 
We  therefore  set  out  to  address  three  goals  collectively  which  are 
summarized  below.  The  results  for  the  first  two  goals  are  reported  in  the 
experimental  section  and  progress  made  in  the  development  of  automated 
scoring  techniques  is  presented  in  the  section  on  software  development. 

1  Compare  eye  movement  metrics  to  performances  on  psychophyslcally 
scaled  workload  tasks.  Establish  the  relationship  between  the  eye 
movement  metrics  and  taskload  and  task  performance. 

2  Examine  the  effects  of  sustained  time  on  task  on  the  eye  movement 
metrics. 

3  Develop  computerized  techniques  and  algorithms  (hardware  and 
software  included)  for  rapid  scoring  and  analysis  of  eye  movement 
Indicants  of  workload  and  vigilance. 

The  ultimate  outcome  of  Phase  II  would  be  the  prototype  of  a  bundled 
transportable  system  to  assess  mental  workload  via  eye  movement  and  other 
bioelectric  measures  as  appropriate.  A  chief  ingredient  in  initial 
development  of  such  a  system  would  be  the  focus  on  rapid  (i.e.,  seconds) 
evaluation  of  the  bioelectric  events  so  that,  when  properly  identified  and 
classified,  such  signals  could  be  fed  back  to  signal  generators. 

III.  EXPERIMENTAL  EFFORT 


METHOD 

SUBJECTS 

College  students  were  recruited  from  the  University  of  Central  Florida 
and  were  paid  $5  per  hour  for  their  participation.  The  subjects  received 
informed  consent  forms  and  were  otherwise  used  in  accordance  with 
established  policies  of  human  use  according  to  nationally  published 
guidelines.  All  subjects  were  instructed  to  perform  low,  medium,  and  high 
difficulty  levels  of  an  auditory  tone  counting  while  eye  movements  were 
recorded  using  electro- oculography  (EOG). 
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EQUIPMENT 


Four  mra  silver/sllver  chloride  electrodes  were  used  for  BCXS  recording. 
Electrode  leads  were  fed  to  three  amplifiers  featuring  characteristics 
suitable  for  EOG  recording.  A  MetroByte  Corporation  DAS-16F  8  channel 
(bipolar)  analog-to- digital  converter  capable  of  sampling  up  to  100,000 
samples  per  second  to  serve  as  the  Interface  to  the  microprocessor.  Three 
channels  were  sampled,  two  for  vertical  eye  movements  and  one  for  horizontal 
eye  movements.  Each  of  the  three  channels  were  sampled  at  256  samples  per 
second  each  (l.e.,  256  samples  per  second  (SPS)  for  the  horizontal  channel, 
256  SPS  for  the  left  eye  vertical  movement,  256  SPS  for  the  right  eye 
vertical  movement). 

The  calibration  board  contains  8  red  LEDs  Imbedded  In  a  four  foot  square 
panel  which  was  painted  black.  The  LEDs  are  controlled  via  software  using 
the  8  digital  I/O  channels  provided  by  the  DAS-16P.  With  the  subject  seated 
5  feet  from  the  center  of  the  board,  and  at  eye  level,  40  degrees  of 
vertical  and  40  degrees  of  horizontal  distance  separate  the  top/bottom, 
left/right  LEDs,  with  10  degrees  of  separation  between  each  LED.  The 
software  calibration  routine  successively  Illuminates  the  calibration  LEDs 
In  the  following  order:  -20  degree  vertical,  -10  degrees  vertical,  hio 
degrees  vertical,  420  degrees  vertical,  ••20  degrees  horizontal,  -10  degrees 
horizontal,  410  degrees  horizontal,  420  degrees  horizontal  and  0  degrees 
(center  LED).  During  calibration  and  while  performing  the  tone  counting 
task,  head  movement  was  restrained  using  a  chin  block. 

PROCEDURE 

Upon  arrival  at  the  experimental  site,  all  subjects  were  briefed 
regarding  the  purpose  and  procedures  of  the  experiment  and  then  were  given 
Informed  consent  forms  to  read  and  sign.  Three  separate  efforts  were 
conducted.  In  the  first  several  subjects  were  exposed  to  the  experimental 
procedures  In  order  to  facilitate  technique  and  to  evaluate  equipments, 
scoring,  etc.  The  main  experiment  employed  20  subjects  but  various 
difficulties  (power  outage,  scheduling,  etc.)  resulted  In  complete  data  to 
be  available  for  15.  The  third  study,  a  probe,  employed  three  subjects  over 
a  one  hour  period  of  recording/testing. 

Vertical  and  horizontal  eye  movement  recordings  were  taken  for  all 
subjects.  Following  the  Initial  briefing,  skin  areas  where  the  surface 
electrodes  were  to  be  applied  were  cleansed  and  then  electrodes  were 
attached  (see  Figure  1)  and  Impedances  checked. 

The  complex  visual  counting  task,  developed  by  Jerlson  (1956),  was 
employed  to  manipulate  workload.  This  task  has  since  been  modified  for 
auditory  presentation  (Kennedy  &  Bittner,  1980)  and  has  been  used  for  many 
years  (Kennedy,  1971).  In  the  first  study,  subjects  (N“15)  were  instructed 
to  perform  low,  medium,  and  high  difficulty  levels  of  the  auditory  tone 
counting  while  eye  movements  were  recorded.  In  the  third  study,  only  the 
two  channel  task  was  employed.  A  microcomputer  was  used  to  administer  a 
quasi  random  series  of  high,  middle,  and  low  pitch  tones„  The  subject’s 
task  was  to  retain  the  count  of  tones  In  working  memory.  Under  the  low 
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workload  (ona  tona)  task  tha  subjact  countad  aacb  oecurranca  of  tha  low 
pitch  tona  and  daprassad  a  kay  aftar  aach  fourth  oecurranca.  Tha  madlum 
workload  task  raqulrad  that  tha  subjact  hold  In  working  memory  both  tha  low 
and  mlddla  pitch  tonas  saparataly  avan  though  thay  ware  prasantad  at 
dlffarant  ratas  (two  tonas).  Low,  mlddla,  and  high  pitch  tones  ware  counted 
and  held  In  memory  separately  for  the  high  workload  condition.  Because 
initial  pilot  tasting  indicated  that  subjects  (N"15)  ware  making  minimal  eye 
movements  during  tona  counting.  It  wan  decided  to  add  tha  a  visual  component 
to  tha  task  so  that  tha  occurrence  of  the  left,  canter,  and  right  LBDs 
(separated  by  10  degrees  horiaontally)  corresponded  to  the  low,  middle,  and 
high  pitch  tonas.  Therefore,  subjects  were  told  to  monitor  tha  lights  in 
addition  to  tha  tonas. 

A  schematic  arrangement  of  tha  test  procedure  for  task  practice, 
calibration,  presentation  of  the  workload  conditions  is  presented  in  Figure 
2.  Prior  to  the  start  of  data  collection,  subjects  received  2.5  minutes  of 
practice  on  each  task.  Following  practice,  each  of  the  three  workload 
performance  tasks  was  administered  twice  for  2.5  minutes  while  eye  movements 
were  measured.  To  counterbalance  order  of  administration,  approximately  1/3 
of  the  subjects  received  each  of  the  three  workload  conditions  first,  second 
or  third.  The  total  session  length  was  approximately  40  minutes. 


Figure  1.  Placement  of  Surface  Blectrodes  for  Bye  Movement  Recordings 
(ground  electrode  attached  to  Inner  left  wrist  not  shown) 
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Figure  2.  Schematic  Arrangement  of  the  Test  Procedure  for  Practice, 
Calibration  and  Test  Administration. 

C  *  Indicates  calibration) 

RB8ULTS  AMD  DISCUSSICM 
PBRFORnANCB  DATA  -  MAIN  STUDY 

Figure  3  shows  performance  averaged  across  15  subjects  for  each  of  the 
three  levels  of  workload.  As  predicted#  performance  (percent  correct)  was 
best  on  the  "low”  workload  task  and  poorest  for  the  "high”  workload  task  and 
these  differences  were  statistically  significant  (p  <  .01).  On  the  high 
workload  task,  no  subject  obtained  a  perfect  (100%  correct)  score  for  the 
five  minutes  of  performance.  On  the  low  workload  task  over  half  of  the 
subjects  obtained  100%  correct  for  the  five  minutes  of  testing.  This 
Implies  that  this  procedure  was  successful  at  the  group  level  In  producing 
task  conurcl  of  workload. 

The  Individual  summary  data  for  each  subject  are  shown  In  Table  1.  in 
general  the  Individual  subject's  performances  reflect  the  group  relations 
although  individual  differences  In  responding  by  some  subjects  implies  they 
may  have  used  different  strategies  In  their  approach  to  the  "high”  versus 
"low”  workloads  of  the  experimental  situation.  This  means  that  while  group 
telations  may  provide  statistically  meaningful  relationships.  In  subject 
dependent  performances  such  as  these,  assessment  of  Individual  bioelectric 
need  to  be  studied  as  well. 

Although  session  length  was  only  40  minutes  and  attempts  were  made  to 
maintain  motivation  for  the  task,  some  of  the  subjects  observed  that  the 
tone  counting  tasks,  particularly  for  the  high  workload  condition,  were 
considered  taxing.  Therefore  we  compared  early  versus  late  performances  for 
evidence  of  a  vigilance  decrement.  Predictably,  tone  counting  performances 
were  generally  better  early  In  the  session,  than  later,  but  these 
differences  were  not  significant  (cf..  Figure  4).  However,  because  of  the 
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Figure  3.  Three  Levels  of  Workload  for  15  Subjects 
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Figure  4.  Performance  Scores  for  the  First  and  Last  Segment  of  Testing 
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TABLE  1.  P«rc«nt  Correct  Response  for  Bach  Subject 
for  the  Low,  Medium,  and  High  Workload  conditions 
Along  with  Summary  Statistics 
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experimental  set  up  and  the  requirement  for  counterbalancing,  minimal  power 
could  be  expected  for  testing  this  relationship,  and  this  may  have  masked 
statistical  verification  of  such  expected  effects.  Elsewhere  for  sustained 
performance  measures,  longer  sessions  have  been  employed  (Kennedy,  1972}  and 
in  those  cases  operators  exhibit  worse  performance  later  in  the  session  than 
earlier.  In  the  longer  sessions  of  the  study  reported  below  we  found  the 
expected  performance  decrement  with  time  on  task. 

DEPENDENT  VARIABLES  -  EYE  MOVEMENTS 

To  facilitate  comparisons  in  evaluating  the  various  eye  movement 
indicants,  we  have  treated  the  one  and  three  tone  monitoring  tasks  as  "low" 
and  "high"  workload  conditions  and  have  summarized  the  findings  in  Figure  5, 
which  shows  the  proportion  of  subjects  who  showed  an  Increase  or  decrease  in 
each  dependent  measure  from  the  low  to  the  high  workload  condition. 
Additionally,  to  index  possible  time  course  eye  movement  metrics,  we  have  made 
summarized  comparisons  for  the  first  five  minutes  of  exposure  versus  the  last 
five  minutes  in  Figure  6  (showing  the  proportion  of  subjects  who  showed  an 
increase  or  decrease  in  each  dependent  measure  from  the  first  to  last  five 
minute  segment  of  testing).  This  approach  to  the  examination  of  the  workload 
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aMtrlcd  ecnblriM  data  ovar  tha  two*  2.5  mlnuta  aaaalona  for  aach  condition.  Xn 
tha  findings  raportad,  tha  sana  ganaral  patterns  ware  generally  obtained  in 
both  2.5  minute  sessions,  and  the  sessions,  therefore,  constituted  a 
replication,  in  future  %iork.  subjects  should  be  tested  over  replications 
conducted  on  separata  days  as  a  check  on  tha  reliability  of  tha  findings. 

tcta  BLIMKS 

Bye  blinks  have  bean  shown  to  be  related  to  workload.  Stem  and  Skelly 
(1984)  assessed  aye  blink,  blink  rata,  and  blink  duration  In  two  simulated 
bomber  missions.  Differing  task  demands  were  reflected  by  blink  rate  and,  in 
addition,  blink  duration  appeared  to  be  affected  by  time  on  task.  In  other 
research,  Bauer,  Ooldsteln,  6  Stern  (1987)  found  that,  in  an  encoding  and 
ratentlcm  task,  blink  rate  was  depressed  following  the  presentation  of  task 
cues,  memory  set,  and  test  stimulus.  The  findings  for  blink  rate  from  the 
present  research  are  summarized  in  Figures  5  and  6  and  presented  for 
Individual  subjects  in  Tables  2  and  3.  In  this  study,  although  blink  rates 
appeared  to  be  reliable  within  a  subject,  blink  rate  per  sp  does  not  appear  to 
be  affected  by  tank  demands  nor  was  blink  rate  affected  by  time  on  task. 


■  mCREASE 


MEASURE 

*Ba8ed  on  test  (l-tall)  for  significance  of  a  proportion. 


Figure  5.  Proportion  of  Subjects  showing  an  Increase  or  Decrease 
in  Bach  Measure  From  Low  to  High  Workload. 
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Figure  6.  Proportion  of  Subjects  Shcetrlng  an  Increase  or  Decrease 
In  Each  Measure  Prom  the  First  to  Last  Segment  of  Testing. 


ThBLB  2.  Humber  of  Bye  Blinks  for  Bach  Subject 
for  the  High  and  Low  Workload  Conditions 


sub 

kggJSi 

High  ¥V 

gub 

Low  VL 

Hlah  VL 

xxb 

52 

22 

dsk 

64 

69 

bms 

e 

14 

ehw 

11 

4 

chb 

24 

13 

bam 

22 

19 

jmc 

63 

155 

egg 

16 

19 

das 

41 

23 

wow 

42 

43 

jat 

13 

8 

dat 

* 

A 

bsw 

138 

179 

wjt 

12 

17 

teh 

10 

9 

*  No  eye  blink  data  collected. 
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TABLE  3.  Number  oC  Bye  Blinks  for  Bach  Subject 
for  the  First  and  Last  Five  Minutes  of  Testing 


Sl!]^ 

5  Mtn 

Last  5  Min 

xxb 

31 

52 

bms 

14 

12 

chb 

29 

24 

Jmc 

63 

68 

das 

23 

41 

Jat 

13 

20 

bsw 

138 

179 

teh 

10 

9 

Sub 

1st  5  Min 

Last  5  Min 

dsk 

69 

73 

ehw 

11 

4 

bam 

21 

22 

egg 

36 

19 

wow 

91 

43 

dat 

* 

* 

wjt 

22 

17 

*  No  eye  blink  data  collected 


FLICK  FREQUENCY 

There  were  wide  Individual  differences  In  the  number  of  saccades  made  by  a 
subject  per  five  minute  epoch  of  workload  performance  and  these  differences 
were  very  reliable.  Frequency  of  saccades  appear  In  Table  4  for  the  two 
workload  conditions.  It  may  be  seen  that  they  do  not  vary  systematically  as  a 
function  of  workload  (see  Figure  5).  However,  Table  5  which  contains 
frequency  of  saccades  for  all  subjects  for  the  first  and  last  five  minutes  of 
testing  shows  that  for  11  out  of  the  IS  subjects  (73%;  £  <  .03),  there  was  a 
decrease  In  frequency  of  saccades  from  the  first  to  the  last  five  minutes  of 
testing  (see  Figure  6).  Frequency  of  saccades  therefore  may  be  related  to 
time  on  task,  and  should  be  followed  as  a  prospective  measure  of  the  vigilance 
decrement  and  sustained  performance. 


TABLE  4.  Number  of  saccades  for  the  Low  and  High  Workload 
Conditions  for  Bach  Subject 


Sub 

Low  VL 

Hloh  WL 

Sub 

Low  WL 

Hlah  WL 

xxb 

267 

195 

dsk 

153 

124 

bms 

41 

73 

ehw 

168 

152 

chb 

311 

306 

bam 

93 

338 

jmc 

198 

222 

egg 

23 

18 

das 

131 

178 

wow 

24 

11 

jat 

15 

16 

dat 

240 

230 

bsw 

221 

84 

wjt 

17 

24 

teh 

64 

92 
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TABLE 

5.  Number 

of  Saccades 
of  Testing 

for  the  First  and  Last  Five 
for  Bach  Subject 

Minutes 

3-.aS-LJ3.ln 

Last  5  Min 

sub 

latL-S-MS 

Last  5  Min 

xxb 

338 

267 

dsk 

124 

134 

bms 

73 

29 

ehw 

168 

152 

chb 

212 

311 

bam 

393 

93 

jmc 

189 

171 

egg 

41 

18 

das 

178 

131 

wow 

47 

11 

jat 

15 

16 

dat 

240 

230 

bsw 

221 

84 

wjt 

17 

24 

teh 

92 

34 

AMPLITUDE  OF  SACCAOES 

In  previous  work<  under  AFOSR  sponsorship  we  (Kennedy,  May,  Jones,  & 
Fowlkes,  1989}  found  the  size  (i.e.,  length)  of  a  saccade  was  related  to 
workload,  such  that  the  mean  range  of  eye  movements  in  the  horizontal  plane 
for  each  of  the  three  workload  conditions  varied  inversely  with  difficulty 
level  of  a  tone  counting  task.  The  present  study  did  not  find  support  for 
this  finding.  It  may  be  seen  in  Figure  5  avef'age  amplitude  of  saccades  was 
shown  to  be  greater  for  the  high  versus  the  low  workload  conditions  (data  for 
individual  subjects  are  presented  in  Table  6).  It  should  be  pointed  out  that 
the  conditions  of  the  present  experiment  were  slightly  different  from  those 
employed  previously  and  entailed  somewhat  greater  visual  articulation  for  the 
high  as  opposed  to  low  workload  conditions.  That  is,  there  may  have  been 
greater  requirements  for  visual  activity  in  order  to  monitor  the  LEDs  which 
were  coincident  with  the  auditory  tones  and  this  may  have  influenced  eye 
movement  activity.  Thus,  under  the  difficult  tone  counting  condition, 
subjects  may  have  been  more  likely  to  monitor  all  three  lights  whereas  In  the 
low  workload  condition  subjects  may  have  been  more  likely  to  monitor  only  one 
LEO  (corresponding  to  the  occurrence  of  the  low  pitch  tone).  Additionally, 
since  performance  data  in  this  experiment  (percent  correct)  were  higher  than 
in  our  previous  research  (possibly  due  to  the  use  of  redundant  information  - 
lights  tones  -  in  this  study),  an  alternative  interpretation  is  that  the 
task  demands  in  the  high  workload  condition  were  not  strong  enough  stimulus  to 
Induce  a  restriction  in  saccades.  These  findings  underscore  the  requirement 
to  specify  very  completely  the  content  of  the  visual  scene  wlien  eye  movement 
related  metrics  are  studied  since  they  are  likely  to  interact. 

Figure  6  (data  for  Individual  subjects  presented  in  Table  7)  summarizes 
the  fact  that  there  was  little  change  (P  <  .15)  in  amplitude  as  a  function  of 
time  on  task;  However,  the  small  apparent  trend  there  was  in  support  of  the 
SLIT  prediction  (l.e.,  a  tendency  toward  larger  saccades  over  time  on  task). 


TABLE  6.  Average  Amplitude  of  saccadeo  (In  degrees)  for  Bach  Subject 
for  the  High  and  Low  Vforkload  Conditions 


Lggja 

HjqlLVL 

xxb 

5.39 

5.33 

bms 

5.70 

5.70 

chb 

3.75 

7.02 

Jmc 

15.28 

10.49 

das 

8.80 

8.68 

Jat 

3.96 

4.14 

bsw 

8.17 

6.97 

teh 

4.08 

3.34 

hSSUSk 

HlatLife 

dsk 

9.21 

8.08 

ehw 

9.70 

11.51 

bam 

6.26 

7.33 

egg 

6.93 

9.72 

wow 

2.39 

8.37 

dat 

13.63 

13.93 

wjt 

8.05 

8.21 

VELOCITY  OP  SACCAOES 

Average  velocity  of  saccades  shown  as  a  function  of  workload  and  time  on 
task  are  shown  in  Tables  8  and  9  and  are  also  summarized  in  Figures  5  and  6. 
From  these  summary  figures  it  may  be  seen  that  the  original  hypotheses  receive 
support.  That  is<  velocity  of  saccades  generally  showed  an  increase  from  low 
to  high  workload  and  a  decrease  from  the  first  to  last  five  minute  segment  of 
testing.  It  is  also  apparent  from  the  figures  that  the  trends  for  velocity 
paralleled  the  trends  seen  for  amplitude.  Because  larger  amplitude  saccades 
are  associated  with  greater  velocities,  we  believe  that  the  trends  for 


TABLE  7.  Average  Amplitude  of  Saccades  (in  degrees)  for  Bach  Subject 
for  the  First  and  Last  Five  Minutes  of  Testing 


sub 

Last  5  Min 

§l!b 

1st  5  Min 

Last  5  Min 

xxb 

6.82 

5.39 

dsk 

8.08 

9.11 

bms 

5.70 

5.23 

ehw 

9.70 

11.51 

chb 

5.64 

3.75 

bam 

6.00 

6.26 

Jmc 

15.28 

11.38 

egg 

10.34 

9.72 

das 

8.68 

8.60 

wow 

6.85 

8.37 

Jat 

3.98 

3.03 

dat 

13.63 

13.93 

bsw 

8.17 

6.97 

wJt 

5.65 

8.99 

teh 

3.33 

4.08 

velocity  could  be  due  to  the  constraints  of  the  task  -  subjects  would  be  more 
likely  to  make  larger  amplitude  saccades,  and  therefore  saccades  of  greater 
velocity,  in  the  high  workload  condition. 


TABLE  8.  Av«rago  Valoclty  of  Saccadas  (in  dagraas/sacond)  for  Each  Subject 

for  tha  High  and  Low  workload  conditions 


S2jb 

Sub 

ftlgt?. 

xxb 

145.28 

154.08 

dsk 

220.29 

207.72 

bms 

179.73 

198.28 

ehw 

237.63 

280.39 

chb 

104.61 

177.58 

bam 

148.62 

185.75 

jmc 

378.91 

276.91 

egg 

182.08 

226.59 

das 

160.47 

181.33 

wow 

73.92 

218.44 

jat 

74.56 

101.53 

dat 

262.47 

267.81 

bsw 

194.19 

175.89 

wjt 

215.30 

255.19 

teh 

108.97 

92.55 

TABLE  9. 

Average  Velocity  of  saccades 
for  the  First  and  Last 

(in  degrees/second)  for  Each  Subject 
Five  Minutes  of  Testing 

Sijb 

Last  5  Min 

Sub 

1st  5  Min 

Last  5  Min 

xxb 

187.80 

145.26 

dsk 

207.72 

221.18 

bms 

198.28 

164.98 

ehw 

237.63 

280.39 

chb 

145.74 

104.61 

bam 

160.85 

148.62 

jmc 

378.91 

274.69 

egg 

232.63 

226.59 

das 

181.33 

160.47 

wow 

193.71 

218.44 

jat 

74.56 

65.93 

dat 

262.45 

267.80 

bsw 

194.19 

175.89 

wjt 

102.63 

255.19 

teh 

92.55 

108.97 

A  NEW  ACCELERATION  METRIC 


Because  of  tha  high  degree  of  association  between  amplitude  and  velocity 
(e.g.f  r  >  .85  in  our  sample),  it  was  deemed  advisable  to  control  further  for 
this  relationship  when  examining  velocity.  Simply  stated,  because  the  orbit 
of  tha  eye  limits  the  extent  over  which  eye  movements  may  be  made,  and  because 
all  eye  movements  follow  the  general  '  form  of  an  acceleration  followed  by  a 
deceleration,  it  was  felt  that  If  there  were  a  triggering  command  signal,  as 
was  hypothesized,  it  would  first  be  necessary  to  control  for  the  physical 
mechanics  of  the  eye  movement  itself.  Therefore,  to  further  analyze  this 
hypothesis,  the  method  of  least  squares  was  used  to  fit  a  linear  regression 
line  to  the  function  between  amplitude  and  velocity  so  that  the  steepness  of 
the  slopes  for  the  high  and  low  workload  conditions  could  be  compared. 
Logically  then,  the  difference  between  the  steepness  of  those  two  slopes  would 
signify  relatively  greater  rates  of  velocity  (i.e.,  acceleration)  between  the 
two  conditions.  A  scatterplot  of  velocity  with  amplitude  is  shown  for  two 
subjects  in  Figure  7  with  the  best  fit  linear  regression  lines  for  the  high 


«nd  low  vwrkload  condltloiui  «lfo  shown.  It  c«n  bo  toon  that  a  stoopor  slopo 
waa  obtalnod  In  tho  high  workload  condition  for  both  aubjocts.  Thia  finding 
was  obtalnod  for  12  of  tho  IS  aubjocta  (80%) »  aa  shown  in  Piguro  5»  auggoatlng 
an  incroaao  in  rata  of  volocit]r  (l.o.,  accoleration)  in  tho  high  workload 
condition. 
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PILOT  STUDT  IN  SUSTkINBD  PBRPORKMfCS 


In  a  pilot  study,  using  only  tho  middle  workload  condition  for  the  tone 
counting  task,  two  subjects  wore  tested  continuously  for  21  minutes  while  eyo 
movements  were  recorded  without  benefit  of  calibration.  The  purpose  of  this 
probe  was  to  determine  Whether  a  longer  period  under  one  task  loading  and 
uninterrupted  by  calibrations  would  result  in  a  vigilance  decrement  and 
thereby  serve  as  a  model  for  sustained  performance  studies.  The  secondary 
purpose  was  to  compare  the  eye  movement  metrics  over  a  protracted  period  of 
time  with  only  one  level  of  workload.  The  primary  purpose  was  largely 
successful.  Data  recording  and  analysis  of  eye  movement  was  accomplished 
without  incident.  The  performance  scores  were  less  clearcut.  For  one  of  the 
subjects,  performance  for  the  first  seven  minutes  was  perfect  (100%  correct), 
while  it  decreased  to  73%  and  80%  correct  for  the  second  and  third  seven 
minute  epochs,  respectively.  Also,  from  the  eye  movement  analysis,  a  similar 
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relationship  between  apparent  workload  (Indicated  by  performance)  and  change 
In  slope  was  observed  as  was  obtained  In  the  workload  study.  Over  this  period 
of  sustained  performance,  there  was  a  systematic  Increase  In  slope  scores  from 
the  first  to  the  last  period,  suggesting  that,  as  alertness  decreased  through 
time  on  task,  there  may  also  be  a  consequent  Increase  In  the  mental  demands  of 
the  task  due  to  the  tedious  nature  of  the  monitoring  activity.  Stated 
differently,  to  the  extent  that  these  slope  scores  Index  task  demands,  time  on 
task  may  have  an  effect  similar  to  adding  additional  channels  of  Information 
to  be  monitored.  There  was  also  a  systematic  decrease  in  the  number  of 
saccades  as  a  function  of  time  on  task,  as  was  found  in  the  workload 
experiment.  The  data  from  the  second  subject,  while  satisfactorily  obtained, 
are  more  difficult  to  reconcile.  For  example,  performance  scores  were  poor 
throughout  the  period  (67%,  60%,  and  70%  correct  for  the  first,  second,  and 
third  periods)  and  showed  negligible  change  over  session.  Although  frequency 
of  saccades  for  the  second  subject  decreased  as  a  function  of  time  on  task, 
slope  scores  systematically  decreased.  Therefore  with  regards  the  slope 
metric  and  time  on  task,  the  findings  from  this  probe  remain  Inconclusive. 

XV.  SOFTWARE  DEVELOPMENT 

As  part  of  the  present  effort  we  sought  to  develop  computer  techniques  and 
algorithms  for  rapid  scoring  and  analysis  of  eye  movement  data.  In  previous 
work  for  the  Air  Force,  we  developed  considerable  amounts  of  customized 
software  which  will  permit  automatic  scoring  of  the  eye  movement  data.  The 
automatic  scoring  program  correctly  Identified  approximately  60%  of  saccades 
and  eye  blinks.  Under  the  present  contract,  we  have  further  refined  this 
software  so  that  it  Is  now  Interactive.  This  was  seen  as  a  necessary 
Intermediate  step  in  the  development  of  a  fully  automated  scoring  system. 

DESCRXPTIOK 

Ve  utilize  FORTRAN-  IV  object  modules  to  aid  In  the  sampling  of 
Analog- to- Digital  channels,  the  programming  of  the  on* board  Real* Time* Clock, 
and  programming  of  the  Digital  I/O  channels.  This  library  is  used  extensively 
in  each  of  the  software  programs  described  below. 

Ve  have  written  software  to  permit  storage  of  analog*  to-dlgltal 
conversions  of  eye  movement  signals  (Including  two* dimensional  recording), 
data  archiving,  and  Immediate  play-*back.  In  addition,  the  software  takes  a 
novice  system  user  through  calibration  techniques  and  actual  use  of  the  system. 

While  It  would  have  been  desirable  to  have  Immediate  on-line  scoring  of 
saccade  lengths,  eye  blinks,  and  saccade  durations,  It  was  found  that  with  the 
computations  Involved,  (even  with  the  high-speed  processor  In  use),  it  was  not 
feasible  without  an  Increased  expenditure  for  more  expensive  and  faster 
hardware. 

Currently,  we  are  able  to  analyze  a  two  and  a  half  minute  session  and  have 
the  results  stored  In  an  Intermediate  file  In  less  than  two  minutes,  with  the 
price  of  fast  DRAM  chips  coming  down  to  a  reasonable  figure,  and  with 
Increased  availability,  most  80386  machines  are  now  within  the  price  range  of 
our  system;  and  we  would  anticipate  that  concurrent  processing  of  the  data 
collected  during  a  session  Is  now  a  fiscally  prudent  possibility. 
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DESCRIPTION  OP  THE  SOFTWARE  MODULES 


All  of  the  executable  modules  are  kept  on  a  single  partitioned  drive  of 
the  computer.  The  source  code  for  the  Essex  developed  modules  are  located  in 
a  sub- directory  as  are  the  libraries  used  for  linkage.  Data  are  collected  in 
the  main  directory,  and  after  an  experimental  run,  are  'squashed',  archived, 
and  stored  In  a  DAT  sub-directory. 

Thera  are  four  modules  ^Ich  need  be  executed  to  cycle  through  an 
experimental  run:  LIVESHCW,  BIOC,  SQUASH  and  SCORE. 

LIVESHOV  Is  run  prior  to  the  start  of  data  recording,  and  after  the 
electrodes  have  been  tested.  Because  there  tends  to  be  a  slight  drift  In  the 
signals,  from  the  beginning  of  a  session  to  the  end,  It  is  necessary  to  ensure 
that  all  signals  start  on  the  zero  line.  LIVBSHOW  enables  to  experimenter  to 
center  each  signal  prior  to  the  execution  of  the  experiment  by  displaying  the 
incoming  signals  from  the  subject  on  the  EGA  monitor. 

Once  the  hardware  has  been  calibrated,  the  run-time  data  recording  task, 
BIOC,  is  executed.  Upon  execution,  the  experimenter  Is  prompted  for  a  three 
letter  subject  Identifier,  most  often  the  first  letter  of  the  subjects  first, 
middle  ond  last  name.  The  experimenter  then  enters  the  order  of  the  tasks  to 
be  performed  by  the  subject,  (i.e.,  high,  middle,  or  low  demand). 

Each  workload  level  is  broken  down  into  two  2.5  minute  sessions.  A 
software  calibration  period  precedes  and  follows  each  2.5  minute  session.  It 
was  decided  to  place  a  calibration  section  at  the  end  of  a  session  due  to  the 
slight  drift  that  occasionally  occurs.  With  a  calibration  period  at  the 
beginning  and  the  end  of  each  session,  automatic  compensation  can  take  place 
in  the  scoring  module.  The  software  calibration  routine  successively 
Illuminates  the  calibration  LEDs  in  the  following  order:  -20  degree  vertical, 
-10  degrees  vertical,  +10  degrees  vertical,  +20  degrees  vertical,  -20  degrees 
horizontal,  -10  degrees  horizontal,  +10  degrees  horizontal,  +20  degrees 
horizontal  and  0  degrees  (center  LED).  The  subject  is  Instructed  to  fixate  on 
the  illuminated  LED  and  one  second's  worth  of  data  is  collected  and  stored  in 
the  data  file. 

The  next  150  seconds  worth  of  data  consist  of  the  actual  eye  movement 
data,  followed  by  9  seconds  of  calibration  data  and,  finally,  the  performance 
scores  for  the  trial  session.  Data  are  stored  as  2's  complement  two- byte 
integer  values,  using  the  upper  12  bits  of  the  word.  The  first  word  of  the 
file  is  data  for  A/D  channel  0,  the  second  is  channel  1  and  the  third  word  is 
channel  2.  Data  are  stored  in  the  above  order  for  each  sample  taken  during 
the  run.  Since  we  sampled  at  256Hz,  one  second  worth  of  data  takes  1536  bytes. 

After  the  last  workload  level  is  completed,  (when  the  experimental  run  is 
over),  the  SQUASH  module  is  executed  to  compress  the  file.  Due  to  the  timing 
of  writing  to  the  disk,  sometimes  zero  data  is  transferred  into  the  file. 
SQUASH  removes  this  non- data  and  effectively  cuts  file  size  by  approximately 
23  percent.  Since  a  complete  testing  session  generates  over  2.5  megabytes  of 
data,  SQUASH  effectively  reduces  this  amount  to  nearly  2  megabytes.  After 
scoring  the  data,  we  further  compressed  the  data  using  PKARC,  an  archiving 
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program,  which  reduced  the  size  of  the  data  files  to  about  1.25  megabytes,  and 
then  we  moved  the  archive  to  the  DAT  subdirectory. 

The  SCORB  module  prompts  the  experimenter  for  the  three  letter  subject  ID, 
the  type  of  graphics  monitor  connected  to  the  computer  (e.g.,  CGA,  BOA.  VGA), 
the  sample  rate  used  during  data  collection,  and  the  number  of  data  points  to 
be  used  for  tentative  saccade  initiation.  After  reading  in  the  initial  nine 
seconds  of  calibration  data,  the  program  becomes  Interactive.  During  this 
experiment,  the  data  was  collected  using  a  sample  rate  of  256Hz.  on  most 
PC's,  this  allows  the  display  of  two  seconds  worth  of  data  on  one  screen.  The 
actual  data  collected  is  then  displayed  on  the  monitor,  scored,  and  results 
stored  in  a  filename  composed  of  the  first  three  Initials  of  the  subject's 
name,  the  testing  session,  and  the  extension  'SAC.  The  SCORE  program 
displays  the  two  vertical  channels  in  dark  blue  and  light  blue,  while  the 
horizontal  channel  is  graphed  in  green.  For  each  frame  of  data  displayed  (2 
seconds),  SCORE  marks  any  blinks  or  saccades  which  the  program  has  attempted 
to  Identify.  Blinks  are  marked  with  a  vertical  light  blue  line,  and  saccade 
beginnings  and  endings  are  designated  by  green  vertical  lines  at  the  beginning 
and  end  of  the  saccades  identified  by  the  program.  At  this  point,  the 
experi.aenter  has  the  option  of  accepting  the  opinion  of  the  program,  or, 
rejecting  the  coraputer  generated  detections  by  either  adding  or  removing 
saccade  and  blink  Identifications.  By  using  the  arrow  keys  on  the  keyboard, 
the  experimenter  can  move  a  vertical  red  line  serving  as  a  cursor  and  by 
positioning  this  cursor  where  desired,  either  add  or  remove  any  blink  or 
saccade  detected.  Table  xx  shows  the  functions  of  the  keypad.  Results  cf  the 
scoring  session  are  stored  in  the  'SAC  file  in  the  following  format; 

event  type,  event  datuml,  event  datum2,  event  datum3,  event  datum4 
where  event  type  would  denote: 

0  -  Initial  calibration  data 

1  -  Computer  detected  blink 

2  -  User  detected  blink 

3  -  Computer  detected  saccade 

4  User  detected  saccade 

5  ■*  Ending  calibration  data 

Depending  upon  thr  type  of  event,  the  data  following  consists  of  varying 
values.  For  blinks  (computer  or  human  detected),  the  first  value  contains  the 
data  point  number  of  the  middle  of  the  blink;  the  other  three  values  are  null 
values  (l.e.,  they  don't  carry  any  meaning).  All  of  the  four  values  for 
saccades  are  meaningful,  on  the  other  hand.  The  first  value  contains  the  data 
point  number  of  the  initiation  of  the  saccade,  the  second  the  data  point 
number  of  the  termination  of  the  saccade,  and  the  third  an!  fourth  contain  the 
voltages  of  the  beginning  and  end  of  the  saccade,  respectively. 

After  the  experimenter  has  scored  the  subject's  data,  another  program  is 
run  which  computes  and  stores  summary  data  for  each  session.  This  file  is 
named  using  the  subject's  three  initials  and  a  'SCO'  extension.  Reported  for 
each  session  are  number  of  eyebllnks,  number  of  saccades,  average  duration, 
and  the  average  velocity  (degrees  per  second)  of  each  saccade.  The  same 
information  is  reported  for  left  and  right  saccades.  The  final  listing 
displays  subject  performance  on  the  counting  task. 


21 


TABLE  10. 

Description  of  Scoring  Keys. 

Left  Arrow 

Move  cursor  left  one  unit 

Shift  Left  Arrow 

Move  cursor  left  10  units 

Control  Left  Arrow 

- 

Move  cursor  left  50  units 

Right  Arrow 

- 

Move  cursor  right  one  unit 

Shift  Right  Arrow 

• 

Move  cursor  right  10  units 

Control  Right  Arrow 

- 

Move  cursor  right  50  units 

Home 

- 

Position  cursor  at  first  position 

End 

- 

Position  cursor  at  li.st  position 

Up  arrow 

- 

Move  horizontal  display  up  10  units 

Down  arrow 

Move  horizontal  display  down  10  units 

S  or  s 

- 

Beginning  of  saccade  conunand  sequence 

S  or  s 

- 

Mark  beginning  of  saccade 

E  or  e 

- 

Mark  end  of  saccade 

R  or  r 

- 

Remove  marked  saccade 

B  or  b 

- 

Beginning  of  blink  command  sequence 

S  or  s 

- 

Mark  blink  at  current  cursor  position 

R  or  r 

- 

Remove  blink  at  current  cursor  position 

Escape 

- 

Terminate  current  command  sequence 

? 

- 

Expand  channel  displays 

1 

« 

- 

contract  channel  displays 

Enter 

Accept  and  store  current  screen 

Cooputnr  saccade  identification  is  comprised  of  three  stages:  the  first 
identifies  saccade  initiation,  the  second  saccade  termination,  and  the  third 
computes  the  voltage  delimited  by  these  points  to  arrive  at  the  size  of  the 
saccade.  To  identify  saccade  Initiation,  the  program  calculates  the  absolute 
difference  in  amplitude  between  five  successive  seunples;  (19.5  ms)  from  the 
horizontal  channel.  If  all  five  differences  exceed  3.66  millivolts  AMD  are  in 
the  same  direction,  a  tentative  saccade  initiation  is  flagged.  Saccade 
termination  is  determined  by  searching  for  five  successive  samples  ivhose 
differences  are  less  than  3.66  millivolts.  If  saccade  termination  criteria  is 
not  net  within  200  ms,  the  tentative  saccade  is  flagged  as  a  false  saccade, 
and  not  included  in  the  data  analysis. 

The  size  of  the  saccade  is  computed  by  taking  the  absolute  difference  in 
amplitude  of  the  end  of  the  saccade  from  the  beginning  of  the  saccade.  This 
results,  in  millivolts,  is  then  multiplied  by  the  scale  factor  obtained  from 
the  calibration  routines  to  arrive  at  the  number  of  degrees  the  eye  moved. 
Duration  of  the  saccade  is  computed  by  simply  subtracting  the  number  of  the 
data  point  which  began  the  saccade,  from  the  ending  data  point  number  and 
multiplying  by  the  known  sample  rate,  which  in  this  case.  Is  256Kz.  For 
example.  If  a  saccade  began  at  data  point  number  1432,  and  ended  at  data  point 
number  1466,  the  duration  of  the  saccade  would  be  34  x  .0039063,  giving  a 
result  of  132.8  milliseconds.  Velocity  of  the  saccade,  in  degrees  per  second, 
is  estimated  from  dividing  the  score  obtained  above  by  the  duration  of  the 
saccade. 
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Ryn  hllnkM  am  relatively  easier  to  detect.  The  vertical  channels  are 
monitored  for  quick,  positive  successive  slopes.  If  five  successive  samples 
show  this  trend,  a  tentative  blink  flaq  Is  set.  The  termination  of  a  blink  is 
detected  by  checking  for  five  successive  samples  with  quick,  negative  slopes. 

V.  CONCLUSIONS 

The  Idea  which  prompted  the  present  research  was  that  biological  events 
may  be  predictive  of  the  attentional  and  task  demands  of  work.  If  these  could 
be  analyzed  in  real  time  and  fed  back  to  the  machine  (or  operator),  a  truly 
biocybernetic  system  could  be  created.  The  purpose  of  Phase  1  was  to 
determine  the  feasibility  of  using  eye  movement  data  to  provide  an  indication 
of  operator  mental  state. 

In  Phase  I  we  set  out  to  accomplish  three  goals: 

1  Compare  eye  movement  metrics  to  performances  on  psychophyslcally 

scaled  workload  tasks.  Establish  the  relationship  between  the  eye 

movement  metrics  and  taskload  and  task  performance. 

2  Examine  the  effects  of  sustained  time  on  task  on  the  eye  movement 
metrics. 

3  Develop  computerized  techniques  and  algorithms  (hardware  and  software 
included)  for  rapid  scoring  and  analysis  of  eye  movement  indicants  of 
workload  and  vigilance. 

For  the  experimental  effort  (goals  1  and  2  above),  we  demonstrated,  first, 
that  task  control  of  workload  had  been  obtained.  Thus,  percent  correct  scores 
obtained  in  this  study  covaried  with  objective  (number  of  tones)  indices  of 
workload.  Further,  eye  movement  metrics  bear  a  relationship  to  these  scaled 
values  so  that  amplitude,  velocity  and  more  strongly,  acceleration  of  eye 
movements  .appear  to  be  related  to  these  objective  controls  of  workload. 
Moreover,  80%  of  the  subjects  show  the  relationship  between  the  acceleration 
metric  and  workload.  The  possibility  exists  that  if  personal  performances 
were  assessed,  it  may  be  possible  to  tune  the  acceleration  metric  in  order  to 
improve  on  its  predictive  power. 

The  case  for  time  on  task  is  somewhat  weaker  because  the  period  of 
sustained  performance  was  short  and  the  change  in  performance  due  to  a 
vigilance  decrement  was  small.  However,  again  the  acceleration  score  shows  a 
change  (it  gets  faster  with  time  on  task)  which  implies  that  it  is  measuring 
some  combination  of  task  loading  plus  time  on  task.  The  frequency  of  eye 
movements  also  decrease  with  time  on  task.  Both  metrics  should  be  followed  as 
a  prospective  measures  of  the  vigilance  decrement  and  sustained  performance. 
A  pilot  study  with  only  two  subjects  and  a  single  workload  task  demand 
produced  mixed  results:  parallel  findings  for  one  subject  who  showed  the 
expected  decrement  in  task  performance  as  a  function  of  time  on  task  and 
contrary  findings  In  a  subject  whose  performance  was  low  and  did  not  degrade. 
These  relations  require  further  study  with  more  subjects,  over  longer  periods 
of  performance  and  who  can  receive  disparate  task  controls  and  demands  and 
mental  work. 
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In  sunnary,  neither  two,  nor  even  fifteen  subjects  may  be  considered 
sufficient  to  do  more  than  gather  hypotheses  for  further  study.  To  some 
extent  the  findings  reported  above  are  encouraging.  Both  eye  movement 
velocity  and  velocity  controlled  for  amplitude  (what  we  called  acceleration) 
appear  to  be  related  to  the  objective  task  demands  on  a  subject.  This  is  now 
an  empirical  outcome  in  this  study  and  follows  from  theory  presented  elsewhere 
and  reported  above.  Thus  when  more  stimuli  are  presented  to  be  retained  In 
working  memory,  this  produces  a  change  in  eye  movement  character  so  that  the 
acceleration  (or  velocity)  is  relatively  greater  in  a  subject  under  ::he  high, 
rather  than  the  low,  workload  condition.  Ve  hasten  to  point  out  that  this 
relationship  is  a  relative  one  and  requires  that  an  individual  first  be 
calibrated  for  his/her  own  "signature”  (obtained  from  a  lower  level  of 
workload)  in  order  to  compare  to  later  outcomes  with  increased  task  demands. 
Ve  believe  that  there  is  considerable  strength  in  this  differential  approach 
to  biomedical  research.  Specifically  it  suggests  that  power  (statistical  and 
otherwise)  can  be  garnered  from  using  each  subject  as  his/her  own  control  and 
then  later  on  customise  procedures  based  on  this  individual  knowledge,  in  the 
present  study,  using  group  specific  findings  (viz.,  means  and  standard 
deviations),  none  of  the  outcomes  would  have  been  revealed. 

Beyond  the  statistically  significant  outcomes  there  are  other 
relationships  which  pose  more  questions  than  they  answer.  For  example,  does 
the  obtained  acceleration  metric  assess  mental  workload,  arousal,  attention, 
working  memory,  lack  of  boredom,  etc.?  Vhy  does  the  acceleration  metric 
correlate  positively  with  velocity  during  workload  and  negatively  during  time 
on  task?  Is  it  possible  (or  likely)  that  time  on  task  is  likely  to  produce  a 
reduction  in  attention  in  some  subjects  and  do  any  of  these  metric  reflect 
that  relationship?  Is  frequency  a  function  of  time  on  task  or  is  it  related 
to  mental/neural  activity  which  is  related  to  performance?  Our  data  imply 
that  certain  neuroelectric  events  follow  somewhat  these  expectations,  but 
there  are  exceptions  and  at  present  the  data  base  from  the  Phase  I  proof  of 
concept  effort  reported  here  is  insufficient  to  rule  cut  other 
interpretations. 

In  terms  of  software  development  (goal  3  above),  we  have  built  upon 
software  developed  under  a  previous  project  for  the  Air  Force.  The  scoring 
routine  has  been  made  interactive  so  that  it  combines  automatic  (computer 
detemined)  and  operator  determined  saccades  and  eye  blinks.  This  step  was 
deemed  important  to  the  goal  of  developing  a  fully  automated  scoring  technique 
for  bioelectric  events. 

The  proposed  Phase  II  is  intended  to  follow  logically  from  the 
accomplishments  of  Phase  I.  If  a  Phase  ll  is  awarded,  we  plan  to  develop  a 
bundled  hardware  and  software  package  which  will  measure  characteristics  of  an 
individual's  eye  movement  patterns  which  predict  the  task  demands  (workload) 
and  separate  them  from  other  eye  movement  characteristics  which  are  related  to 
time  on  task.  If  such  information  were  available  with  sufficient  precision, 
it  would  be  possible  for  the  USAP  to  take  the  output  of  an  individual's 
interactions  with  displayed  information  and  modify  the  display  so  the  system 
will  better  accommodate  the  individual's  needs.  Pending  success  of  Phase  IX, 
a  Phase  III,  funded  by  internal  research  and  development  capital  from  Essex, 
will  prepare  a  prototype  tool  to  be  marketed  in  the  private  sector. 
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